Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JENKINS-68116 Make event thread pool size customize-able #124

Merged

Conversation

timja
Copy link
Member

@timja timja commented Mar 25, 2022

See https://issues.jenkins.io/browse/JENKINS-68116

Slow processing of events is causing delays of over an hour for our users to have their builds / pushes picked up

Currently increases thread pool size from 10 -> 30 and adds a system property to make it customise-able

Last changed in #61

I've ran this for four hours on a busy controller and never saw the queue grow above 4 or the thread pool get maxed out with running threads (highest I saw was ~27).

Running this in script console shows the status:

$ jenkins.scm.api.SCMEvent.executorService

Result: java.util.concurrent.ScheduledThreadPoolExecutor@23c09aa1[Running, pool size = 30, active threads = 20, queued tasks = 0, completed tasks = 4029]

Related PR: jenkinsci/branch-api-plugin#304

@timja timja marked this pull request as ready for review March 25, 2022 14:50
@timja timja requested a review from a team March 25, 2022 14:51
@timja
Copy link
Member Author

timja commented Mar 25, 2022

I'm going to do some more log searching now that I've increased the amount of logs to see if there's anything useful.
I'm wondering if we have some repositories with a lot of branches that take awhile to process.

@timja
Copy link
Member Author

timja commented Apr 4, 2022

@jtnord @jenkinsci/scm-api-plugin-developers I've reduced the size to 10 for now by default but making is customiseable to at least unblock me.

are we able to integrate just this change?

FWIW this doesn't scale right now very well, see https://issues.jenkins.io/browse/JENKINS-68116?focusedCommentId=423884&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-423884

@car-roll car-roll requested a review from jtnord April 7, 2022 04:42
@timja timja changed the title JENKINS-68116 Increase event thread pool size and make customize-able JENKINS-68116 Make event thread pool size customize-able Apr 9, 2022
@timja
Copy link
Member Author

timja commented Apr 11, 2022

ping @jtnord

@timja timja force-pushed the JENKINS-68116-make-thread-count-customise-able branch from f7fbf52 to a611e79 Compare April 26, 2022 13:38
@timja
Copy link
Member Author

timja commented Apr 26, 2022

Ping @jtnord / @jenkinsci/scm-api-plugin-developers

I've reduced our folder count from 68 to 6 which seems to be performing better but I would still like to be able to configure this above 10. It doesn't make sense for a 10 repository instance to have the same processing capacity as an org with over 1500 repositories

@timja timja force-pushed the JENKINS-68116-make-thread-count-customise-able branch from a611e79 to 62eb573 Compare April 26, 2022 13:41
@jtnord jtnord merged commit 1622544 into jenkinsci:master Apr 26, 2022
@timja timja deleted the JENKINS-68116-make-thread-count-customise-able branch April 26, 2022 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants